Make coverage reporting actually run, and send it to Codecov - #964
Conversation
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
|
Status on the red checks — one was a real bug, now fixed; the other two are an artifact of enabling Codecov.
Every test in the file errored. Only the minimum-versions job caught it, since it pins pytest 7.0 and the nose-style hooks were removed in pytest 8 — the other jobs run a newer pytest and passed. It is now imported as
Both compare against
Nothing regressed — coverage of I have deliberately not changed the targets to make these go green. Generated by Claude Code |
The `coverage` job has been a no-op. `report-coverage` merges the matrix
artifacts server-side but never downloads them into the workspace, so
every command in it ran against an empty directory:
+ python -Iim coverage combine
No data to combine
+ python -Iim coverage report --fail-under=80
No data to report.
The job still passed, because `-Iim` includes `-i`: after each command
exits non-zero, Python drops into an interactive interpreter, reads EOF
from the empty stdin, and exits 0. So the 80% gate never once fired.
Replace it with explicit steps that download the `coverage-*` artifacts,
combine them, and publish the result to Codecov. The checkout the job
already does turns out to be load-bearing: merging the Windows data with
the POSIX data relies on `relative_files`, and coverage only maps a
recorded path onto a canonical one when that file exists on disk.
The floor is set to 78%, just under the 80.26% a single run measures
today, so that it stays a backstop rather than a tripwire. Codecov's
`project` status, at `target: auto`, is what ratchets coverage up.
Also move `branch` into `[tool.coverage.run]`. It was passed as
`--cov-branch` from the hatch script, which left the `coverage` CLI
invocations in CI measuring something subtly different from the pytest
run that produced the data.
Codecov needs `CODECOV_TOKEN` in the repository secrets; without it the
upload falls back to tokenless, which is rate-limited. `fail_ci_if_error`
is off until that secret exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qMpoL5cL6W4qCeCdGm26w
Codecov already merges every upload it receives for a commit, so the artifact round-trip and the combine job were doing work the service does anyway. Each matrix job now uploads its own coverage.xml, tagged with a flag, which also gets the per-OS and per-version breakdown that a single combined upload cannot show. `coverage xml` writes the `filename` attributes with forward slashes regardless of platform, so the Windows uploads line up with the rest without the path mapping the combined data file needed. Drop `after_n_builds`. It was correct at 1 for a single upload; with one upload per matrix job it would have to track the matrix size, and would silently report partial coverage the first time someone adds a Python version without bumping it. `wait_for_ci` (true by default) already holds the comment and the statuses until CI has finished. This gives up the `--fail-under` gate that ran in the combine job. Codecov's `project` status covers it, and is the stricter of the two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qMpoL5cL6W4qCeCdGm26w
…nent `manager.py` and `sphinxdoc.py` were both at 0% -- 117 statements that no test ever touched. Neither needs much of a harness: the config manager is JSON on disk, and the Sphinx extension turns out to import no Sphinx at all, so a stub with an `add_object_type` method is enough to exercise `setup()`. Both are now at 100%, branches included, along with the gaps in `descriptions.py`, `getargspec.py`, `sentinel.py` and `bunch.py`. The library goes from 80.26% to 84.59%. For the components, coverage has to measure the test suite as well -- a component can only report on files that are in the report. Codecov then splits the two apart, since blending them gives a number (89.25%) that mostly tracks how much of the test code runs rather than how much of the library is covered. Measuring from the repository root, rather than `source = ["traitlets", "tests"]`, is deliberate. `coverage xml` writes each path relative to its source root, so with two roots both `config/__init__.py` files land in the XML under one name. Today they are empty and nothing collides; the first line of code added to `tests/config/__init__.py` would silently merge the two on Codecov's side. One root keeps the paths distinct. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qMpoL5cL6W4qCeCdGm26w
pytest before 8.0 treats a module-level `setup` as an xunit setup hook and
calls it once per test with the test module as its argument, so importing
the extension's `setup` into the test module's namespace turned every test
in the file into an error:
AttributeError: module 'tests.config.test_sphinxdoc' has no attribute
'add_object_type'
Only the minimum-versions job caught it, since it pins pytest 7.0 and the
nose-style hooks were removed in pytest 8. Verified against pytest 7.0.1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qMpoL5cL6W4qCeCdGm26w
`application.py` sat at 76% with ~113 uncovered lines, nearly all of them the help machinery: emit_alias_help, emit_flag_help, emit_options_help, emit_subcommands_help, emit_help, emit_description, emit_examples and the print_* wrappers around them, plus start_show_config, load_config_environ, boolean_flag, get_config and launch_instance. Most of that is not actually untested. test_help_output and friends drive the application through `check_help_output`, which spawns a subprocess, and coverage does not follow the child. So the lines run, but nothing records it. These tests call the emitters in process instead, one method at a time, which measures them and pins the generator API that downstream applications use. 40 tests, taking application.py from 76% to 94%. The remaining misses there are error paths that re-raise after logging, and would need a deliberately malformed alias or flag to reach. tests/test_typing.py is now omitted. mypy type-checks it and pytest-mypy-testing never executes the bodies, so all 454 lines read as uncovered -- it is a fixture for the type checker, not dead code, and counting it only depressed the number. The library goes from 84.59% to 87.07%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qMpoL5cL6W4qCeCdGm26w
2c4d4b6 to
f1eed9f
Compare
os.environ upper-cases every key on Windows, so HELPAPP__Foo__name arrives
as HELPAPP__FOO__NAME. load_config_environ splits the trait name off the
end and assigns it unchanged, and Config rejects a key beginning with an
uppercase letter unless the value is another Config:
ValueError: values whose keys begin with an uppercase char must be
Config instances: 'NAME', DeferredConfigString('from-the-environment')
This is not specific to the names in the test. Every trait name is
upper-cased the same way, so no environment variable can set any trait on
Windows -- the "Warning, case sensitive!" note in the method is
understating it. That behaviour predates this branch; the tests merely
reached code that nothing had reached before.
Skipping keeps the tests honest about the limitation without asserting
that the broken behaviour is correct, and Linux and macOS still cover the
method. Fixing load_config_environ is a behaviour change, and belongs in
its own pull request.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qMpoL5cL6W4qCeCdGm26w
|
The Windows jobs went red on The three new tests for it failed on Windows 3.10, 3.13 and 3.14 with:
_, *path, key = k.split("__")
section = new_config
for p in path:
section = section[p]
setattr(section, key, DeferredConfigString(v))
This predates the branch. Nothing in the suite reached Generated by Claude Code |
Summary
The
coveragejob onmainhas never reported anything. Its log:Two separate bugs in
jupyterlab/maintainer-tools/.github/actions/report-coverage@v1:actions/upload-artifact/merge) but never downloads them into the workspace, so everycoveragecommand ran against an empty directory.python -Iimincludes-i. Each command exits non-zero, Python then drops into an interactive interpreter, reads EOF from the empty stdin, and exits 0 — so the job went green and the--fail-under=80gate never fired once.This PR replaces that job with per-matrix-job uploads to Codecov, adds Codecov components, and fills in the library's largest coverage gaps.
CI wiring
.github/workflows/tests.yml— each matrix job uploads its owncoverage.xmlviacodecov/codecov-action@v5, taggedflags: <os>-<python-version>. Codecov merges the uploads for the commit, so the artifact round-trip and the wholecoveragejob are gone.codecov.yml— new.project: autowith a 0.5% threshold,patch: 80%, comment only when coverage changes.pyproject.toml—branch = truemoved into[tool.coverage.run](it was--cov-branchon the hatch script only, so any barecoverageinvocation measured something different), and--cov-report xmladded tocov:test.README.md— Codecov badge..gitignore—coverage.xml.Coverage
traitlets/config/application.pyconfig/manager.pyconfig/sphinxdoc.pyutils/descriptions.pyutils/getargspec.pyutils/sentinel.pymanager.pyandsphinxdoc.pyhad no tests at all — 117 statements. The Sphinx extension imports no Sphinx, so a stub with anadd_object_typemethod exercisessetup().application.py's ~113 uncovered lines were nearly all the help machinery, and most of it was not untested:test_help_outputand friends go throughcheck_help_output, which spawns a subprocess that coverage doesn't follow. The lines ran; nothing recorded it. The new tests callemit_alias_help,emit_flag_help,emit_options_help,emit_subcommands_help,emit_help,emit_description,emit_examples, theprint_*wrappers,start_show_config,load_config_environ,boolean_flag,get_configandlaunch_instancein process — which measures them, and pins the generator API downstream applications build on. What still isn't covered there is error paths that log and re-raise.93 new tests, 675 total, passing on both current pytest and the pinned minimum (7.0.1).
tests/test_typing.pyis omitted from measurement: mypy type-checks it and pytest-mypy-testing never executes the bodies, so all 454 lines read as uncovered. It's a fixture for the type checker, not dead code.Components
component_managementdefinestraitlets(traitlets/**) andtests(tests/**), each with its ownprojectstatus. A component can only report on files in the report, so coverage measures the test suite too — a test file with uncovered lines is a test that never runs. That blends the headline number upward, which is why the split matters:traitletsis the one to watch.Coverage is measured from the repository root rather than
source = ["traitlets", "tests"]. That's deliberate:coverage xmlwrites each path relative to its source root, so with two roots bothconfig/__init__.pyfiles land in the XML under the same name. They're empty today and nothing collides — the first line of code added totests/config/__init__.pywould silently merge the two on Codecov's side.Verified, not assumed
state: complete, 22 sessions, 40 files on an earlier revision. The 40 vs 43 files measured locally is exactly the three empty__init__.pyfiles undertests/, whichcoverage xmlomits as zero-statement.coverage xmlnormalises thefilenameattribute to forward slashes on every platform (coverage/xmlreport.py:190,207), which is why the Windows uploads line up with the rest.codecov.ymlvalidates againstcodecov.io/validate, components included.0ef86a9.Notes for review
CODECOV_TOKENis not set. Uploads take the tokenless path, which works but is rate-limited and less reliable on fork PRs. Worth adding, then flippingfail_ci_if_error: true.after_n_buildsis deliberately absent. With one upload per matrix job it would have to equal the matrix size, and would silently report partial coverage the first time someone adds a Python version without bumping it.wait_for_ci(defaulttrue) already holds the comment and statuses until CI finishes.--fail-undergate is gone with the combine job; Codecov'sprojectstatus replaces it.codecov/projectchecks compare against a November 2022 baseline (1077cfb), the last commit on main carrying a report. That comparison is measuring two different things and resolves itself once this lands and main gets a fresh report. Details in the thread below.coveragewas a required status check in branch protection, the check no longer exists and will need removing — I can't read branch protection settings from here.